Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2003 


Supplementing  Classroom  Training  with  a  Distributed  Refresher  Intelligent 

Tutoring  System 


Randy  Jensen 
Richard  Stottler 
Stottler  Henke  Associates,  Inc. 
San  Mateo,  CA 
Jensen@StottlerHenke.com 

Stottler@StottlerHenke.com 


Dr.  Sharon  Garcia 
Air  Force  Research  Laboratory 
Brooks  AFB,  TX 
Sharon.Garcia@Brooks.af.mil 

Milton  Waddell 

Air  Force  Command  &  Control  Warrior  School 
Hurlburt  Field,  FL 
Milton.Waddell@Hurlburt.af.mil 


ABSTRACT 

The  training  regimen  for  personnel  entering  an  Aerospace  Operations  Center  (AOC)  must  cover  a  complex 
organizational  structure  with  complex  information  flows,  both  horizontally  and  vertically.  Training  and  orientation 
includes  classroom  lectures  on  many  different  specialty  areas  within  the  AOC,  supplemented  with  exercises  offering 
limited  practice  in  greater  depth  with  some  of  these  areas.  Recent  exercises  have  revealed  a  need  for  further 
refresher  training,  particularly  in  focus  areas  within  the  AOC  that  involve  complex  decision-making  based  on  AOC- 
specific  processes,  as  opposed  to  general  operational  knowledge  or  experience.  This  paper  presents  a  proof-of- 
concept  system  developed  to  demonstrate  web-based  refresher  training  for  the  Master  Air  Attack  Plan  (MAAP) 
focus  area  within  the  AOC.  The  benefits  of  refresher  training  are  well-documented,  for  the  flexibility  they  provide 
trainees  in  reviewing  concepts  on  their  own  from  different  locations  and  “just-in-time”  when  given  a  new 
assignment.  However,  distributed  refresher  training  without  an  intelligent  tutoring  component  can  amount  to  little 
more  than  an  online  textbook.  By  implementing  Intelligent  Tutoring  System  (ITS)  methods  in  the  distributed 
setting,  trainees  can  benefit  from  scenario-specific  and  student-specific  feedback  in  response  to  their  performance  on 
simulated  operational  exercises.  The  system  described  in  this  paper  is  based  on  a  limited  principle  hierarchy 
developed  and  integrated  with  an  online  simulation.  This  simulation  models  an  existing  software  environment  for 
the  MAAP  decision-making  process.  The  student  is  given  an  introductory  briefing  on  a  targeting  and  resource 
allocation  task,  and  the  student’s  subsequent  performance  in  developing  an  attack  plan  and  information  coordination 
plan  is  evaluated  in  terms  of  the  system’s  principle  hierarchy.  The  system  supports  an  iterative  training  loop,  where 
the  student  can  revisit  the  exercise  to  modify  earlier  planning  decisions,  and  receive  updated  feedback. 
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INTRODUCTION 

The  training  regimen  for  personnel  entering  an 
Aerospace  Operations  Center  (AOC)  must  cover  a 
complex  organizational  structure  with  complex 
information  flows,  both  horizontally  and  vertically. 
Training  and  orientation  includes  classroom  lectures  on 
many  different  specialty  areas  within  the  AOC, 
supplemented  with  exercises  offering  limited  practice 
in  greater  depth  with  some  of  these  areas.  Recent 
exercises  have  revealed  a  need  for  further  refresher 
training,  particularly  in  focus  areas  within  the  AOC 
that  involve  complex  decision-making  based  on  AOC- 
specific  processes,  as  opposed  to  general  operational 
knowledge  or  experience.  This  paper  presents  a  proof- 
of-concept  system  developed  to  demonstrate  web- 
based  refresher  training  for  the  Master  Air  Attack  Plan 
(MAAP)  focus  area  within  the  AOC. 

The  benefits  of  refresher  training  are  well-documented, 
for  the  flexibility  they  provide  trainees  in  reviewing 
concepts  on  their  own  from  different  locations  and 
“just-in-time”  when  given  a  new  assignment. 
However,  distributed  refresher  training  without  an 
intelligent  tutoring  component  can  amount  to  little 
more  than  an  online  textbook.  By  implementing 
Intelligent  Tutoring  System  (ITS)  methods  in  the 
distributed  setting,  trainees  can  benefit  from  scenario- 
specific  and  student-specific  feedback  in  response  to 
their  performance  on  simulated  operational  exercises. 

The  system  described  in  this  paper  is  based  on  a  limited 
principle  hierarchy,  which  was  developed  and  integrated 
with  an  online  simulation.  This  simulation  models  an 
existing  software  environment  for  the  MAAP  decision¬ 
making  process.  The  student  is  given  an  introductory 
briefing  on  a  targeting  and  resource  allocation  task,  and  the 
student’s  subsequent  performance  in  developing  an  attack 
plan  and  information  coordination  plan  is  evaluated  in 
terms  of  the  system’s  principle  hierarchy.  The  system 
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supports  an  iterative  training  loop,  where  the  student  can 
revisit  the  exercise  to  modify  earlier  planning  decisions, 
and  receive  updated  feedback. 

AOC  TRAINING  REQUIREMENTS 

The  Air  Force  Command  and  Control  Training  and 
Innovation  Group  is  responsible  for  providing  training  for 
Aerospace  Operations  Center  (AOC)  operational 
personnel.  However,  the  AOC  is  very  complex,  with 
complex  processes,  complex  information  flows  (both 
horizontal  and  vertical),  and  dozens  of  personnel.  Tire 
current  training  regimen  consists  of  instructor-led  courses 
combined  with  practice  in  independent  sessions  where 
students  gain  specific  practice  with  examples.  Each  trainee 
must  understand  the  overall  structures  and  process  of  the 
AOC;  how  their  specific  role  fits  into  both  of  these;  how  to 
assess  the  importance  and  route  incoming  information  in 
the  context  of  these  information  flows,  organizational 
structures,  and  current  high-level  and  lower  level  goals  and 
objectives.  In  addition,  they  must  understand  the 
principles  behind  and  how  to  use  the  application  software 
and  systems  and  how  to  correctly  apply  all  of  this 
knowledge  to  make  correct  decisions  in  operational 
situations.  These  decisions  may  include  both  decisions 
relating  to  information  flow  or  application  use  as  well  as 
decisions  relating  to  a  particular  area  of  expertise,  such  as 
defensive  tactics  or  logistics. 

The  Joint  Aerospace  Command  and  Control  Course 
(JAC2C)  currently  given  at  the  C2  Warrior  School 
includes  many  sections  with  overview  level  orientation  in 
many  different  specialty  areas  within  the  AOC.  Classroom 
lectures  are  supplemented  with  exercises  offering  limited 
practice  in  greater  depth  with  some  of  these  areas.  Tire 
instructor-led  training  is  an  essential  introduction  to  the 
AOC,  its  processes,  and  information  flow  within  the  AOC. 
But  recent  BLUEFLAG  exercises  have  revealed  a  need  for 
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further  refresher  training,  particularly  in  focus  areas  within 
the  AOC  that  involve  complex  decision-making  based  on 
AOC-specific  processes,  as  opposed  to  general  operational 
knowledge  or  experience.  In  particular,  the  Master  Air 
Attack  Plan  (MAAP)  process  within  the  AOC  Combat 
Plans  division  has  been  an  area  targeted  for  supplemental 
tools  and  training,  due  to  both  the  complexity  and 
criticality  of  the  MAAP  process,  and  also  the  need  for 
students  to  develop  effective  cognitive  models  for  how  to 
use  information  effectively  while  preparing  a  MAAP. 

There  are  many  examples  in  the  MAAP  process  where 
competing  principles  must  be  resolved  in  the  process  of 
allocating  resources  to  targets.  If  there  are  two  groups 
of  targets,  one  near  and  one  far  with  respect  to  an 
available  base,  and  the  near  group  has  mostly  low 
priority  targets  and  the  far  group  has  high  priority 
targets,  then  the  planner  needs  to  be  flexible  in  the 
application  of  heuristics  for  target  allocation.  It  is 
appropriate  to  consider  other  issues  as  well  that  may 
resolve  such  a  dilemma.  For  example,  would  a  strike 
package  planned  for  the  far  target  group  encounter  any 
known  surface-air  missile  (SAM)  sites  along  the  flight 
path?  Perhaps  those  SAMs  should  be  addressed  first  in 
a  preliminary  wave  before  dealing  with  the  far  targets. 
Are  there  any  high  priority  targets  in  other  nearby 
target  groups  that  could  be  combined  in  a  package  that 
strikes  the  near  group?  What  is  the  nature  of  the 
desired  effect  for  the  far  targets?  If  successive  strikes 
will  be  depending  on  the  achievement  of  the  objectives 
associated  with  the  far  targets,  then  the  high  priority 
level  associated  with  these  targets  may  outweigh  the 
additional  difficulty  or  risk  in  reaching  them.  For  each 
of  these  different  factors,  the  student  must  not  only  be 
able  to  consider  how  they  should  impact  a  planning 
decision,  but  also  know  to  ask  these  questions  and 
know  how  to  find  the  answers. 

The  MAAP  Toolkit  is  a  software  suite  recently 
developed  to  accelerate  MAAP  build  time,  simplify  the 
revision  process,  reduce  the  number  of  required 
personnel,  and  integrate  different  data  sources.  These 
are  all  critical  benefits,  and  have  all  been  quantified  at 
JEFX  trials;  for  example,  a  30%  reduction  in  MAAP 
production  time  was  measured.  Therefore,  while  this 
tool  provides  an  important  value  for  facilitating  the 
process,  instructors  still  noted  a  deficiency  in  trainees 
when  it  came  to  their  cognitive  models  of  the  MAAP 
process  itself,  and  a  lack  of  understanding  of  key 
concepts  involved  in  best  practice  decision-making  for 
optimizing  a  Master  Air  Attack  Plan.  Deficiencies 
range  from  overlooking  regular  planning 
considerations  like  allocating  critical  support  assets 
such  as  refueling  nodes  or  suppression  of  enemy  air 
defense  (SEAD)  with  the  construction  of  a  strike 


package,  to  over-tasking  strike  assets,  or  failing  to 
consider  new  information  from  Intelligence 
Surveillance  and  Reconnaissance  (ISR),  or  the 
Battlefield  Coordination  Detachment  (BCD).  Thus,  the 
current  training  regimen  for  students  headed  for  service 
in  a  MAAP  team  needs  to  be  augmented  with  refresher 
training  specifically  focusing  on  the  MAAP  process,  so 
that  students  can  get  more  practice  and  feedback  with 
exercises.  They  need  to  gain  familiarity  not  only  with 
the  operational  concepts,  but  also  with  the  structure  of 
the  MAAP  team  and  the  inputs,  outputs,  and  process  in 
which  the  operational  concepts  are  best  applied. 


DETERMINING  THE  COMPLEMENTARY 
ROLES  FOR  CLASSROOM  AND  DISTRIBUTED 
REFRESHER  TRAINING 

Because  the  current  training  for  the  MAAP  process 
occurs  within  the  context  of  a  broad  overview 
perspective,  there  is  a  clearly  identifiable  need  for  more 
specific  practice  and  feedback-oriented  training. 
Current  instructors  and  program  managers  alike  agreed 
that  refresher  training  would  complement  the  existing 
training  effectively,  especially  if  delivered  as  a 
distributed  trainer,  by  providing  the  opportunity  for 
more  in-depth  practice  with  MAAP  exercises,  but  with 
more  flexible  time  requirements  and  virtually  no 
additional  facility  requirements.  With  the  opportunity 
for  more  practice  in  operational  exercises,  students  can 
develop  better  cognitive  models  of  the  MAAP  process, 
which  leads  to  better  decision-making.  Although  the 
refresher  trainer  will  have  several  key  distinctions  from 
the  existing  broad  classroom-based  curriculum,  it  can 
also  leverage  much  of  the  existing  embedded 
knowledge  and  courseware.  Reusable  materials 
include  the  Pacifica  conflict  scenario,  sample  inputs 
used  for  classroom-based  exercises,  such  as  JIPTLs 
(Joint  Integrated  Prioritized  Target  Lists),  resource 
lists,  realistic  force  availability  breakdowns, 
weapons/loads  guides,  weather  reports,  and  so  forth.  In 
addition,  the  instructors  themselves  represent  a 
valuable  asset  for  constructing  both  the  exercises  for 
the  system  and  the  evaluation  criteria  for  assessing 
student  outputs. 

With  the  objective  of  providing  refresher  training  as 
opposed  to  initial  training,  the  system  employs 
simplified  exercises  to  target  specific  principles.  This 
is  most  suitable  to  the  distributed  refresher  training 
setting,  which  is  generally  most  effective  as  a  platform 
for  repetitive  practice  with  numerous  exercises  of 
shorter  duration  (1  hour  or  less).  One  of  the  important 
benefits  of  making  distributed  training  available  is  the 
flexibility  it  offers  trainees  in  the  sense  of  providing  the 
opportunity  for  access  from  different  locations,  and  for 
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different  durations  of  time.  So  a  critical  difference 
from  classroom  training  when  it  comes  to  instructional 
authoring  is  the  specific  requirement  that  refresher 
training  exercises  be  designed  for  compatibility  with 
this  kind  of  usage  profile,  which  means  shorter 
duration.  In  order  to  accomplish  this,  each  exercise 
needs  to  abstract  certain  concepts.  This  is  necessary  to 
enable  students  to  focus  on  key  concepts  and  that  they 
can  then  be  evaluated  on  accordingly.  Although  the 
operational  exercises  for  the  refresher  ITS  are  designed 
for  abstraction  and  brevity,  there  are  no  intrinsic 
technical  obstacles  to  exercises  of  longer  duration,  so 
future  systems  may  include  a  minority  of  such 
exercises,  depending  on  instructor  input  on  what  is 
appropriate. 


MODELING  THE  DECISION-MAKING 
ENVIRONMENT  AND  PRINCIPLES 

With  the  MAAP  process  domain,  there  are  different 
kinds  of  skills  and  knowledge  that  need  to  be  applied. 
One  skill  type  relates  to  decision-making  concepts 
involved  in  planning;  for  example,  grouping  targets  by 
geography  and  priority,  and  resolving  resource 
conflicts.  Another  skill  type  involves  domain-specific 
knowledge;  for  example,  an  A- 10  can  carry  an  MK-84 
SCL,  but  an  F-15E  cannot.  It  is  important  to  note  that 
much  of  the  second  type  of  domain-specific 
information  is  automatically  provided  within  the 
environment  of  the  MAAP  Toolkit.  This  has  two 
consequences  for  the  refresher  trainer.  First,  it 
establishes  a  dividing  line  for  the  kinds  of  concepts  that 
need  to  be  addressed  in  refresher  training.  Since  the 
second  type  of  information  is  readily  available  from 
software  tools  and  resource  sheets,  it  was  left  outside 
the  scope  of  the  instructional  concepts  to  be  addressed 
in  the  system.  Second,  it  presents  an  embedded 
knowledge  problem  for  the  refresher  ITS.  For 
example,  if  a  student  using  the  ITS  attempts  to  create 
an  F-15E  mission  carrying  an  MK-84,  either  the  system 
should  give  a  warning  or  simply  disallow  this  action. 
The  consequence  of  this  requirement  is  that  the  system 
needs  to  maintain  a  basic  database  of  domain 
knowledge  that  can  be  used  on  first  pass  evaluations  of 
student  actions.  A  preliminary  rough  version  of  this 
database  is  implemented  for  numerous  aircraft  and 
weapons  in  the  proof-of-concept  ITS. 

The  ITS  uses  exercises  based  on  the  Pacifica  scenario, 
which  is  also  used  for  training  in  other  military 
applications  and  courses.  It  has  been  observed  in  Air 
Force  training  that  students  respond  best  to  training 
scenarios  that  can  be  briefed  in  detail  to  provide  an 
element  of  reality,  which  essentially  establishes  a  sense 
of  “what  they  are  fighting  for.”  The  Pacifica  scenario 


is  commonly  used  because  students  are  often  already 
familiar  with  the  broad  parameters  of  the  conflict,  and 
instructors  can  make  use  of  this  familiarity  in  the 
introductory  briefing  that  establishes  the  setup  for  a 
particular  exercise.  Likewise  in  the  distributed  setting, 
by  using  the  Pacifica  scenario,  the  spinup  time  for  each 
exercise  provided  by  the  ITS  can  be  minimized,  and 
students  can  get  to  the  exercises  quickly.  The  exercise 
scenario  in  the  proof-of-concept  system  was  designed 
with  the  input  of  instructors  at  the  C2  Warrior  School 
who  have  developed  similar  examples  for  classroom 
instruction  and  practice. 

Broadly  speaking,  the  inputs  to  the  student  in  the  ITS 
consist  of  a  geographic  map,  a  merged  collection  of 
resource  sheets,  and  a  target  list.  The  student’s  output 
is  a  simplified  version  of  a  Master  Air  Attack  Plan  - 
essentially  a  list  of  missions  and  packages  developed 
for  the  day’s  ATO.  The  student  constructs  a  solution  in 
the  web-based  simulation  environment,  which  is  a 
simplified  version  of  the  operational  MAAP  Toolkit 
environment.  Although  the  ITS  is  not  a  primary  visible 
component  of  the  user  interface,  in  contrast  with  the 
simulation  environment,  the  automated  evaluation  it 
performs  on  student  outputs  is  a  major  part  of  the 
software  functionality  in  this  system.  Once  the  student 
has  completed  the  preparation  of  the  outputs  for  a 
given  operational  exercise,  the  ITS  evaluation  is 
triggered  with  a  simple  single  button. 

The  ITS  employs  a  target  list  acting  as  a  simulated 
JIPTL,  which  in  the  real  AOC  comes  to  the  MAAP 
team  from  the  Guidance  Apportionment  and  Targeting 
(GAT)  team,  but  was  developed  for  this  proof-of- 
concept  by  instructors  from  the  C2  Warrior  School  at 
Hurlburt  Field.  In  the  simulation  interface,  the  student 
can  create  either  strike  or  support  missions.  In  the 
operational  exercises,  the  majority  of  the  student 
actions  involve  specifying  strike  missions,  allocating 
necessary  support,  and  resolving  resource  conflicts 
associated  with  this  part  of  the  planning  process.  An 
example  instructional  principle  is  to  understand  when 
and  how  to  allocate  Air  Refueling  for  missions  in  part 
of  a  strike  package.  For  all  the  principles  in  the 
hierarchy,  the  student  actions  in  the  simulation 
environment  are  monitored  so  that  customized 
feedback  can  be  provided. 


AUTOMATED  EVALUATION,  FEEDBACK, 
AND  REMEDIATION  THROUGH 
OPERATIONAL  EXERCISE 

The  ITS  is  structured  around  an  operational  exercise 
format  which  includes  a  set  of  preliminary  briefings  to 
establish  the  broad  scenario,  the  specific  target,  the 
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tactical  objectives  and  commander’s  intent,  and  any  relevant  intelligence  reports. 


Yourtask,  in  support  of  the  tactical  task,  is  to  assemble  a  strike  package  for  a  single  target,  the  Bishop  Airfield,  a 
Priority  A  target.  The  two  DMPIs  you  will  plan  forthis  part  of  the  package  are: 

1 .  Fixed  wing  aircraft  in  the  open  on  the  airfield 
2  Rotary  wing  aircraft  in  the  open  on  the  airfield 

Intelligence 

Previous  intelligence  reports  have  picked  up  an  emitter  from  one  enemy  SA-2  site,  about  15  miles  away  from  the 
Bishop  Airfield.  The  airfield  is  expected  to  have  2  to  4  MIG-29  Fulcrum  flying  defensive  patrol 


Briefings  MAAP  Workspace  Evaluation 


Scenario  Briefing 
Tactical  Briefing 


Target  Briefing 
Intel  Update  Briefing 


Figure  1:  Target  Briefing  Screen 


Figure  1  shows  the  Target  Briefing  for  the  Bishop 
Airfield  scenario.  These  briefings  do  not  only  establish 
context  for  the  planning  task,  they  also  provide 
essential  information  that  should  be  taken  into  account 
in  the  MAAP  that  the  student  will  develop.  For 
example,  the  briefings  provide  information  about 
known  enemy  SAM  sites,  which  the  student  must 


address  in  the  MAAP,  typically  with  some  form  of  air 
suppression. 

Once  the  student  is  ready  to  begin  the  air  attack 
planning  process,  the  next  step  is  to  enter  a  MAAP 
Workspace  tab  in  the  web-based  environment.  The 
workspace  is  the  setting  for  the  entry  of  all  strike  and 
support  missions. 
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<9  Strike 
C  Support 

C  Information  Flow  and  Coordination 


Update  Intelligence 


Target  Name: 

Suggested  SCL: 

Aircraft  Type: 

Resource  Availability 

Bishop  Airfield 

JDAM 

A/C  Type 

SCL#  Base  Name  Availability 

Desired  Effect: 

GBU  10 

B-52 

12  j  Guam  7 

80%  Aircraft  destroyed 

F-15E 

5  Michael  AA...  0 

Create  New  Mission 
TOT: 


Save  to  Mission 


Mission 

Msn#  MsnType  TOT/Task 


1 58993  Strike_ Bishop  Airfield  ]TOT:1030  Michael  AAF,  NE 


Time 


Base 


I  Type A/C  I  #A/C  1  SCL  I  SCL* 


B-52 


JDAM 


12 


Figure  2:  Strike  Planning  Interface 


For  a  strike  mission,  the  student  can  select  a  target 
either  visually  on  a  geographical  map  or  textually  from 
a  target  list  table.  The  student  can  zoom  in  or  out  on 
the  map  image,  with  targets  maintaining  their  relative 
positions.  When  the  student  selects  a  target  on  the 
map,  the  corresponding  row  in  the  target  list  is 
highlighted,  and  likewise  in  the  opposite  sequence. 
With  a  target  selected,  the  student  can  choose 
individual  DMPIs  (Desired  Mean  Points  of  Impact) 
associated  with  the  target,  and  see  the  desired  effect  for 
each,  as  would  normally  be  provided  by  GAT  on  a 
target  nomination  list.  For  each  DMPI,  there  is  a  list  of 
suggested  SCLs  (Standard  Conventional  Loads),  again 
based  on  GAT  input,  from  which  the  student  can  make 
the  decision  about  assigning  resources  and  weapons  for 
the  current  DMPI.  This  decision  is  based  on 
knowledge  about  available  resources  in  terms  of 
aircraft,  and  the  locations  of  the  bases  or  assets  from 


which  the  aircraft  originates.  A  limited  resource  sheet 
database  is  implemented  with  the  system,  so  that  when 
the  student  selects  a  potential  SCL  from  which  to 
construct  a  mission,  only  the  suitable  and  available 
aircraft  appear  in  the  Aircraft  Type  list.  Similarly,  for 
each  aircraft  in  this  list,  only  the  bases  that  have  the 
selected  aircraft  will  appear  in  the  Resource 
Availability  list.  Once  a  DMPI,  SCL,  Aircraft,  and 
Resource  are  selected,  the  student  may  specify  a  TOT 
(Time  On  Target)  and  create  a  strike  mission  with  these 
parameters.  The  mission  is  added  to  the  Missions  list 
at  the  bottom  of  the  environment,  and  will  ultimately 
be  evaluated  by  the  ITS  as  part  of  the  student’s  output. 

The  process  is  somewhat  different  for  support 
missions,  as  they  can  involve  a  variety  of  support  tasks, 
such  as  escort,  refueling,  or  Suppression  of  Enemy  Air 
Defense  (SEAD),  which  may  not  be  associated  with 
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specific  DMPIs  or  weapons.  These  missions  are 
specified  in  a  separate  environment  that  pops  lip  upon 
selection  of  the  Support  Task  button,  and  they  are 
subsequently  added  to  the  student’s  list  of  outputs  for 
evaluation. 

Additionally,  there  is  a  third  category  of  student  output, 
which  involves  the  explicit  designation  of  information 
coordination  plans.  The  student  is  provided  a  checklist 


for  designating  coordination  with  liaison  elements  from 
other  forces  and  with  other  sources  of  data  such  as 
collection  management. 

Once  the  student  has  completed  preparing  the  outputs 
for  a  given  operational  exercise,  the  ITS  evaluation  and 
debriefing  is  triggered  by  clicking  on  the  Evaluate 
button. 


Briefings  MAAP  Workspace  Evaluation 
Principles  to  Test 

X  1.  Review  latest  Intelligence  Information. 

X  2.  If  required,  create  sorties  to  destroy  or  disrupt  SAM  threats 
X  3.  Update  targeting  to  deal  with  new  threats  reported  in  latest  intelligence. 

I  S  4.  If  needed  allocate  missions  for  Air  Refueling  with  long-range  aircraft 
X  5.  If  needed  allocate  missions  for  Air  Refueling  with  short-range  aircraft 
IS  6.  If  required  allocate  assets  for  sweep  and/or  escort  with  bombers. 

?  7.  If  required  allocate  assets  for  sweep  and/or  escort  with  fighters. 

S  8.  Create  sufficient  strike  sorties  to  effectively  complete  the  task  at  the  assigned  target. 
I S  9.  Don't  over-allocate  assets  to  targets/escort/SAM/etc. 

X  10.  Include  specification  of  desired  effect  at  SAM  sites  in  SEAD  tasking 
X  11.  Appropriate  information  coordination  and  collection  management  on  strike  package. 


Evaluate 


Refresh 


Feedback 

It  is  important  to  ensure  you  have  the  latest  intelligence,  especially  when  strike  planning  for  targets  where  there  are  previously  known 
surface-to-air  threats  Any  information  about  newly  detected  emitters,  or  new  locations  for  known  SAM  sites,  must  be  considered  in  assembling 
a  strike  package  so  that  proper  SEAD  assets  may  be  employed. 


Figure  3:  ITS  Feedback  Screen 


Figure  3  shows  the  feedback  screen  provided  by  the 
ITS  after  the  student  has  assembled  a  MAAP  and 
requested  initial  evaluation.  A  simple  report  card 
format  is  provided,  with  a  summary  of  the  relevant 
decision  making  principles,  annotated  to  reflect  the 
student’s  success  or  failure  with  each.  Successfully 
understood  principles  are  marked  with  a  green 


checkmark,  and  failed  principles  are  marked  with  a  red 
X.  In  some  cases,  there  may  be  additional  information 
related  to  a  principle,  which  is  constructive  to  report  to 
the  student,  without  necessarily  returning  a  pass  or  fail 
judgment,  in  which  case  the  principle  is  marked  with  a 
question  mark.  The  student  can  receive  more  details 
by  clicking  on  an  individual  principle  and  reading  the 
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information  provided  in  the  Feedback  region.  The  ITS 
is  designed  around  an  iterative  model,  where  students 
can  update  their  air  attack  plans  based  on  the  feedback 
received,  and  return  successive  times  for  additional 
evaluation.  The  feedback  provided  by  the  ITS 
intentionally  avoids  giving  specific  solutions,  as 
opposed  to  general  suggestions  about  key  issues  for  the 
student  to  consider. 

For  each  operational  exercise,  the  student  outputs  are 
evaluated  with  respect  to  a  collection  of  fragmentary 
good  and  bad  answers  that  have  been  developed  with 
the  input  of  instructors.  In  a  simple  example,  the 
student  may  have  forgotten  to  allocate  missions  for  Air 
Refueling  for  one  or  more  of  the  aircraft  included  in 
the  strike  package.  In  a  more  complicated  example,  a 
student  may  have  allocated  weapons  for  a  DMPI  as 
part  of  a  given  strike  package  that  meet  the  GAT 
recommendations  and  correct  prioritization  order.  This 
would  match  with  the  corresponding  elements  of  a 
good  solution  provided  by  instructors  for  the  ITS. 
However,  suppose  the  student  used  assets  from  a  base 
that  is  distant  from  the  specified  targets,  when  aircraft 
from  another  base  closer  to  the  target  could  have  been 
used  with  the  same  weapons.  This  could  be  especially 
bad  if  the  longer  flight  path  increased  exposure  to 
surface-air  missiles  or  other  threats,  or  required 
unnecessary  use  of  refueling  nodes  as  a  result.  This 
would  match  with  another  fragmentary  bad  solution, 
and  trigger  system  feedback  accordingly.  Thus,  in  the 
debriefing,  the  student  receives  an  appropriate 
combination  of  positive  and  negative  feedback,  which 
directly  responds  to  performance  in  the  exercise. 

The  ITS  is  completely  web-based,  and  can  be  viewed 
over  a  browser  on  Java-enabled  machines.  The 
architecture  primarily  involves  an  interface  applet 
which  communicates  with  a  server-side  database  and 
ITS.  The  server-side  database  includes  the  domain- 
related  information  such  as  resource  compatibility  (for 
example,  which  aircraft  can  carry  which  loads),  the 
data  for  the  operational  exercises  (target  lists,  resource 
availability),  a  small  amount  of  courseware, 
information  about  the  student  which  is  maintained  in  a 
student  model,  and  a  principle  hierarchy  based  on  the 
cognitive  model  of  the  domain  and  the  small  set  of 
principles  selected  for  treatment  in  the  system.  The 
ITS  also  resides  on  the  server,  initially  triggered  by  a 
message  from  the  applet  and  then  after  completing  a 
review  of  student  outputs,  it  provides  a  debriefing  back 
to  the  applet’s  browser  interface.  The  proof-of-concept 
system  serves  to  demonstrate  a  browser-based 
environment  where  domain-specific  knowledge  can  be 
embedded  to  simulate  the  many  data  sources  that  serve 
as  inputs  to  the  MAAP  process,  which  is  a  traditional 
information  flow  problem  within  the  AOC. 


LESSONS  LEARNED  AND  FUTURE 
APPLICATIONS 

As  a  proof-of-concept,  this  system  was  developed  with 
the  intention  of  demonstrating  that  meaningful 
instructional  value  can  be  realized  in  the  distributed 
refresher  setting,  when  coupled  with  ITS  technology. 
Although  there  is  an  abstraction  process  involved  in 
presenting  complex  problems  in  a  web-based  platform, 
it  is  clearly  possible  to  isolate  key  instructional 
principles  in  a  practice  environment  and  provide 
individualized  performance  feedback  on  these 
principles.  The  research  effort  also  extended  into 
design  work  for  a  more  full-scale  refresher  ITS  for  the 
AOC  domain,  much  of  which  consisted  of  developing 
more  detailed  specifications  for  components  of  the 
proof-of-concept  system. 

In  order  to  tailor  the  course  of  study  to  the  individual 
student,  the  ITS  keeps  a  model  of  each  student  who 
uses  the  ITS.  The  student  model  contains  the  student’s 
actions  and  decisions  during  different  scenario 
exercises,  the  principles,  models,  processes,  skills, 
procedures,  and  techniques,  which  have  been 
presented,  and  those  that  have  been  mastered  based  on 
performance  in  exercises.  The  set  of  principles, 
models,  processes,  procedures,  factors,  and  strategies 
referenced  in  the  solutions  of  problems  the  student  has 
solved  successfully  represent  the  student's  acquired 
skills.  Based  on  the  pattern  of  unsatisfactory 
performance  on  exercises,  a  set  of  topics,  principles,  or 
combinations  of  them,  can  be  developed  which  form  a 
hypothesis  as  to  what  information  the  student  does  not 
understand.  For  example,  the  student  might  not 
understand  the  MAAP  principle,  “Include  electronic 
warfare  measures  as  needed  in  attack  packages.” 
Based  on  this  hypothesis  and  the  exercises  solved 
incorrectly,  similar  examples  can  be  shown  to  increase 
the  student's  understanding.  For  example,  given  the 
previous  MAAP  principle  in  which  the  student  is 
deficient,  the  ITS  might  present  an  exercise  where  the 
enemy  has  radar  nodes  that  should  be  jammed  with 
electronic  countermeasures  as  part  of  a  strike  package. 
It  may  be  appropriate  to  require  the  student  to  re¬ 
experience  some  of  the  course  topic  material  as  well, 
perhaps  to  a  greater  level  of  detail.  Based  on  this 
hypothesis,  a  new  set  of  exercises  can  be  generated  for 
testing  the  success  of  the  remediation.  A  course 
instructor  or  manager  to  monitor  the  student’s  progress 
through  topics  and  performance  in  exercises  can  also 
reference  the  student  model.  The  student  model  will 
reflect  the  skills,  knowledge,  and  error-rate  of  the 
student.  The  student  model  evolves  in  size  and 
complexity  as  the  skills  and  knowledge  of  the  student 
increase. 
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After  re-testing,  the  success  of  the  remedial  instruction 
can  be  gauged.  By  varying  the  type  of  this  remedial 
instruction,  the  most  appropriate  instructional 
technique  for  this  student  can  be  inferred. 
Furthermore,  the  ITS  can  infer  different  “best” 
instructional  techniques  for  different  types  of  principles 
and  can  thus  infer  different  best  instructional 
approaches  for  different  types  of  tasks. 

The  system  design  takes  a  very  general  view  of  what  is 
meant  by  the  term  “Principle.”  It  is  basically  anything 
which  must  be  understood  and  applied  during  cognitive 
decision-making.  In  the  case  of  AOC  operations, 
individual  principles  might  represent  that  the  student 
understands  the  importance  of  various  types  of 
information  from  different  groups.  For  example,  one 
particular  principle  might  represent  that  a  student 
understands  that  the  attack  package  to  be  prepared  in 
an  exercise  is  a  part  of  a  large  push,  and  thus  the 
strikes’  TOTs  should  match  up  with  the  guidance  from 
the  strategy  cell  with  regard  to  serial  or  parallel  attacks. 
This  is  a  specific  component  of  an  information  flow 
model.  However,  an  AOC  individual  must  also 
understand  how  to  make  tradeoffs  and  deconfliction 
within  the  collection  of  targets  in  the  individual’s  own 
package  command,  and  this  would  be  represented  and 
assessed  by  several  individual  principles.  Thus  very 
different  types  of  knowledge  can  be  represented  in  the 
ITS  and  tracked. 

The  evaluation  component  of  the  system  design  is  one 
of  the  most  complex.  In  traditional  ITSs,  an  expert 
system  is  applied  to  the  same  problem  as  the  student, 
and  their  actions  are  compared.  Unfortunately,  the 
creation  of  the  expert  system  is  difficult  or  impossible 
even  for  a  skilled  knowledge  engineer  in  most  military 
domains.  In  our  approach,  the  correct  actions  (or  an 
exercise-specific  way  to  derive  them)  are  stored  with 
the  exercise,  so  the  Evaluate  module  need  only 
compare  the  student’s  actions  to  these. 

One  refinement  of  this  method  is  to  store  correct  and 
common  incorrect  scenario  fragments  entered  by  the 
instructor  for  each  exercise.  Each  component  of  the 
fragment  is  annotated  with  an  explanation  as  to  why 
that  solution  is  correct,  partially  correct,  or  wrong 
along  with  the  principle  mastery  (good  or  bad)  that  is 
indicated  by  the  trainee’s  decision  if  it  matches  that 
component  of  the  fragment.  After  the  trainee  enters  a 
solution,  it  is  compared  to  each  of  the  fragments  and 
the  closest  matching  ones  are  used  as  a  basis  for 
assembling  the  debriefing  and  the  list  of  principles 
mastered,  as  illustrated  by  the  student’s  solution.  For 
illustration,  consider  a  simple  exercise  with  two  target 
groupings  J  and  K.  The  J  targets  are  all  B  priority,  and 
the  K  grouping  includes  mostly  A  priority  targets. 


Suppose  further  that  the  GAT  recommendations  list 
GBU-12s  as  the  best  choice  weapons  for  both  sets  of 
targets,  but  that  only  one  aircraft  capable  of  delivering 
GBU-12s  is  available  for  this  wave  within  this 
package.  If  the  student  allocates  the  aircraft  and  the 
GBU-12s  to  the  J  targets,  then  this  qualifies  as  passing 
the  principle  associated  with  using  the  most 
recommended  weapon  for  that  particular  target  set. 
However,  the  student  also  fails  the  principle  of 
allocating  available  assets  first  for  the  highest  priority 
targets.  Triggering  the  different  pass  and  fail  principles 
depends  on  the  representation  of  solution  fragments  for 
the  operational  exercise  so  that  the  system  can  perform 
matching  during  the  evaluation  process.  Authoring  the 
Evaluation  Knowledge  simply  requires  entering  these 
annotated  exercise  solution  fragments. 

Many  of  the  task  areas  within  the  AOC  could  make  use 
of  the  instructional  tools  described  here  with  little 
transition  cost.  The  design  and  implementation  work 
in  this  system  represents  a  preliminary  step  for  future 
implementation  of  more  comprehensive  refresher 
training,  both  for  the  MAAP  process  and  for  other 
areas  within  the  AOC. 
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